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I, Terry Potter, hereby declare as follows: 

1 . I am a co-inventor of the data correlation and analysis tool described and 
claimed in the above-identified patent application serial number 10/821,750 
("the '750 application"). I earned my PhD in System Science with a major in 
Computer Science from SUNY Binghamton in 1987. 

2. I have reviewed U.S. Patent No. 5,517,251 (Rector) and conducted a search for 
information regarding IFFPHYS described in Rector at column 6, lines 37-46. 
Enclosed herewith as Exhibit A is an article I found: Rector, David M. et al, 
"Continuous Image and Electrophysiological Recording with Real-Time 
Processing and Control," METHODS 25, 151-163 (2001). 

3. An associative mapping as described in the '750 application has significant 
advantages when stored as a table. The associative mapping can be accessed by 
the content of any selected rows or columns. Additional rows or columns can 
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be added at any time to provide additional response variables or to include an 
additional stimulus stream. For example, "newly created variables 106 can be 
one variable, such as an average, or a multitude of variables, such as a set of 
values of how far each process variable is from the mean." ('750 application, 
page 9, lines 25-27) Also the time slices of a video signal taken at a different 
angle from the original video signal could be added onto the table and thus be 
related to the response variables. Searching through any given row or column 
for variables meeting a search criteria can be accomplished very quickly. 

4. The IFFPHYS file format of Rector, on the other hand, is simply a flat file for 
continuous image and electrophysiological data recording. The data is mixed or 
interleaved with the video signal producing one combined signal in the 
IFFPHYS file format. 

5. Searching through an IFFPHYS file would require reading through the entire 
signal video and data, since they are mixed or interleaved into a single file. 

6. Once the file has been created, Rector shows no accommodation for mixing in 
additional data. Indeed, space for data is limited to that which is available in the 
video signal, whereas the associative mapping table format of the '750 
application allows unlimited additions of data. Should the computer memory be 
exhausted, more memory can be added and the table format permits the addition 
of more data. 

7. The IFFPHYS file format mixes a plurality of data signals into one video file. 
The data is inserted into the vertical blanking interval and other available spaces 
in the video signal. Given the format of the IFFPHYS file a second video signal 
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does not make sense and Rector does not suggest any more than the one video 
signal. 

8. I hereby declare that all statements made of my own knowledge herein are true 
and all statements made on information and belief are believed to be true; and 
further that these statements were made with the knowledge that willful false 
statements and the like so made are punishable by fine or imprisonment, or 
both, under Section 1001 of Title 18 of the United States Code and that such 
willful false statements may jeopardize the validity of the application in 
connection with which this declaration is being submitted to the Patent and 
Trademark Office, or any patent issued thereon. 
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Continuous Image and Electrophysiological Recording 
with Real-Time Processing and Control 

David M. Rector and John S. George 1 



Collecting continuous video together with multichannel electro- 
physiological data and other experimental modalities requires high 
bandwidth and storage capacities, as well as accurate synchroniza- 
tion to detect correlations between different recorded events. Of- 
ten, experiments are highly complex, with many variables requiring 
immediate analysis and feedback during the course of the experi- 
ment. In addition, output channels require real-time control with 
high time resolution. We have explored several approaches to a 
system that can perform the above functions. The design of our 
system considered a number of issues, including time intervals 
between control and acquisition events, longest continuous re- 
cording period, data transfer bottleneck considerations, file archiv- 
ing and format, and real-time display and processing. To demon- 
strate the system, we describe an experiment for characterizing 
rapid evoked scattered light changes in neural tissue, in vivo, using 
simultaneous electronic image acquisition and electrophysiologi- 
cal recording. © 2001 Elsevier Scigncg 



The ability to record from large neural populations 
simultaneously is necessary to assess how neurons be- 
have in a network. Although multiple electrode tech- 
niques have grown rapidly from the use of a few elec- 
trodes to a hundred or more electrodes in a tight array 
(1), electrical measurements using such arrays continue 
to be invasive and of limited spatial resolution. In prin- 
ciple, a tightly integrated multiplexed amplifier system 
might provide a practical cost-effective data system; 
however, existing systems allow acquisition of at most 
a few dozen parallel channels. 

From very early in the history of neuroscience investi- 
gation, optical microscopy has been used to study the 

1 To whom correspondence should be addressed at Biophysics Group, 
P-21, MS-D454, Los Alamos National Laboratory, P.O. Box 1663, Los 
Alamos, NM 87545. Fax: (505) 665-4507. E-mail: jsg@lanl.gov. 
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structure and function of neural tissue. Optical meth- 
ods have a number of advantages; they can sample 
the relevant range of spatial and temporal scales, and 
provide information on microanatomy as well as bio- 
chemical and physiological processes. A number of in- 
vestigators have observed reflected light changes from 
activated cortical surfaces and exploited the phenome- 
non to generate spatial organization maps of columnar 
structures in visual cortex that complement electro- 
physiological recording (2-4). Others have visualized 
sensory activation patterns in human temporal cortex 
(5) and rodent sensory cortex (6, 7). 

Although optical measurements can resolve single 
action potentials in the giant squid axon and crab nerve 
(8, 9), these demanding measurements have previously 
only been accomplished with sensitive single-channel 
technologies. Most investigators interested in network 
dynamics have used dyes sensitive to membrane poten- 
tial or intracellular ion concentrations, and employed 
photodiode array detectors (8, 10). Recent studies use 
fast CCD cameras with less toxic dyes (11); however, 
associated instrumentation remains expensive with 
limited functionality. Delivery of dyes is difficult and 
photoreaction may cause damage. 

Although imaging is a useful technique, a network 
in its basic form contains patterns of communication 
between neurons in a network. The network has input 
and output projections that collect information and in- 
fluence structures outside the network (Fig. 1). To fully 
understand the behavior of a given network, we must 
control input to the network or perturb neurons within 
the network directly and simultaneously record effector 
responses of the imaged network using multichannel 
analog techniques. There are many basic requirements 
that a data acquisition and control system must meet 
to perform these experiments. A fast, sensitive video 
camera is needed to image the network activity. Digital 
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and analog outputs are used to perturb the network 
in a systematic way and a multichannel analog input 
system is required to record the effector responses. 
These components need to be synchronized, with a well- 
defined time granularity archived onto a mass storage 
device, and displayed and processed in real time so that 
digital and analog controls can respond to the patterns 
observed within and external to the network, effectively 
forming a closed loop. 

Recording video and many electrophysiological chan- 
nels simultaneously poses a significant challenge to 
data acquisition systems. We were unable to identify a 
commercial product to perform fast video digitizing that 
would also acquire analog channels for electrophysiol- 
ogy. Some systems specialize in digitizing a few chan- 
nels very quickly (1 to 4 channels at 100 MSPS, 8 bits), 
or in digitizing a large array of separate input channels 
(256 channels) but at relatively slow rates (e.g., an ag- 
gregate rate of 1 MSPS). Also, many multichannel sys- 
tems use multiplexing switches with very slow on/off 
times (greater than 1 juls), thus reducing the system's 
overall ability to acquire multiplexed channels at 
high rates. 

Several products claimed to deliver the performance 




FIG. 1. A typical imaging paradigm includes a sample neural popu- 
lation with input fibers and output fibers (or effectors). An optimal 
system for recording the behavior of the neural population includes 
electrical recording electrodes, an imaging device, and detection of 
output pathways from the population. Dynamic feedback and pertur- 
bation of the system can be achieved through a stimulus paradigm. 
The system for performing these tasks needs to accept input data 
and simultaneously to control output parameters such as CCD cam- 
era timing signals and stimuli. An effective system would have inte- 
grated acquisition and control as well as real-time processing for on- 
line display and feedback. 



we required; however, the actual limitations were dis- 
appointing, and the systems could not perform the de- 
sired tasks. Rapid and very long data streams were 
difficult to handle. Data could not be simultaneously 
acquired and displayed at maximum acquisition rates. 
Limited on-board buffer memory made long continuous 
data streams impossible. Additionally, video acquisition 
and high-resolution experimental control required sep- 
arate systems or boards which were difficult or impossi- 
ble to synchronize. 

For these reasons, we developed methods for multi- 
plexing electrical and video information. The easiest 
method involved building a simple switching circuit so 
that a portion of each video line was replaced with 
concurrent electrophysiological values, resulting in a 
signal that could be captured with a video digitizer and 
parsed for video and electrophysiological data (13, 14) 
(U.S. Patent 5,517,25 1). This procedure worked well for 
synchronizing slow electrical signals (<200 Hz) with 
image data at real-time video frame digitizing rates 
(60 fps); however, this method had limitations for 
higher-performance frame capture (>100 Hz) and 
faster electrical signals (>1000 Hz). 

A variety of commercial video digitizers provide the 
least expensive and easiest method for digitizing video 
signals multiplexed with analog channels. Unfortu- 
nately, there are also several limitations with inexpen- 
sive, consumer-grade digitizing boards. Many manufac- 
turers claim 8-bit digitizing, but store only 5 or 6 bits of 
gray-scale information in memory. Low dynamic range 
simplifies video compression video for home movies, but 
precludes detection of small signal changes. We have 
found that any type of lossy compression scheme such 
as "mpeg" creates artifacts in low-contrast images when 
they are subtracted or ratioed. 

Both consumer and scientific grade boards have a 
limited throughput for continuous, full, and uncom- 
pressed video frames (e.g., 3 fps). Some scientific grade 
boards with PCI interfaces can stream video into moth- 
erboard memory, but usually only for a limited amount 
of time and at the expense of all other processor opera- 
tions. Further, the NTSC standard limits the total 
frame rate to 60 fps and a maximum dynamic range of 
8 bits. To reduce the total data rate and to free resources 
for other tasks, a few boards allow for selection of a 
subregion of interest. Despite the inability to multiplex 
electrophysiology with a standard video signal, com- 
mercially available systems remain inadequate for 
high-performance recordings. 

We have recently demonstrated dynamic imaging of 
intrinsic optical signals in vivo (12). In addition to slow 
changes associated with hemodynamic activation, we 
have observed fast light scattering changes associated 
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with the physical processes of neurophysiological acti- 
vation. The basic requirements for a system to record 
these signals (outlined in Table 1), include 100 to 1000- 
fps video resolution, 12 to 16 bits of dynamic range, 
16 to 256 electrophysiological channels at 1 kHz each 
channel, and 25-ns timing granularity to control the 
CCD camera in real time. The continuous data stream 
produced is typically 1 to 64 MB per second, limited 
mainly by the rate at which we can archive the data. 
On-line display typically shows images and histograms 
for one of every three video frames. On-line time-trig- 
gered averaging captures every sequence generated. We 
use inexpensive but high-performance CCD chips for 
our miniature imagers and have developed a cost-effec- 
tive, high-performance system for video, electrophysiol- 
ogy, and other data acquisition and for instrument and 
experiment control. 

The digitizing system we developed performs all of 
the operations described above. Data inputs include 1 
to 32,000 electrophysiological channels and one or more 
video signal inputs at 100 to 10,000 fps. The aggregate 
data throughput rate is currently limited to 32 MS/s 
(64 MB/s); however, migration from the PCI interface 
to the faster Compact-PCI or PCI-X bus would double or 
quadruple that value. The digitizer, analog multiplexer, 
and video control outputs are controlled synchronously 
by a field programmable gate array (FPGA). This device 
is configured as a state machine that executes a se- 
quence of microcode instructions optimized for data ac- 
quisition and control. Data are double buffered so that 
when one buffer is full, the data banks are switched 
and the host downloads the data. The host archives the 
data to a large hard disk, performs some calculations on 
the data (e.g., image sequence averaging and histogram 
generation), and displays the data. The software is writ- 
ten so that separate tasks are divided into separate 
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functional processes, with interprocess communication 
through network socket protocols. This allows different 
tasks to be performed by one CPU, multiple CPUs, or 
many computers connected through a network. 



DATA ACQUISITION SYSTEM 

We developed an inexpensive solution to meet our 
data acquisition and control needs (Fig. 2). A host com- 
puter with PCI interface capabilities, fast display, and 
mass storage devices is configured with a high-perform- 
ance operating system such as Linux. Other operating 
systems including Windows, MacOS, and OS2 are too 
slow and cumbersome to handle high-volume continu- 
ous data streams along with on-line display. Software 
is written in "C" using Unix socket protocols to make 
data available to internal processor tasks, or through 
a fast ethernet connection to other computers in a 
multitasking environment. Built-in DMA can transfer 
data from the acquisition board to the host computer 
through the PCI bus at 132 MB/s or through a Compact 
PCI bus at 528 MB/s or higher. DMA transfers occur 
in parallel with CPU operations, freeing the CPU for 
simultaneous data processing. 




Electrophysiological channels 
1 ilect ro physiological channel 

Video pixels per frame 
Bit resolution 

Aggregate data throughput 



100-1000 Hz 



10,000-250,000 
16 bits 

100-5,000 fps 
64 MB/s 



a Data throughput is calculated from electrophysiological and video 
parameters. The typical performance is achieved with a multiproces- 
sor computer using RAID archive hardware. 



FIG. 2. Block diagram of the acquisition and control system. A host 
mm pule i' running I he LINUX opera I i tig system communicates with 
separate processes t h rough network sockets for archiving onto amass 
storage device, on-line processing and display, and fast ethernet con- 
nect ion lo remote computers. An interlace hoard containing a I'd 
interface, double-buffered memory banks, and an Analog Devices 
SHARC DSP chip plugs into the PCI slot of the host computer. As 
many as six remote boards can be connected to the main interface 
board, each of which uses another SHARC DSP and a loop controller 
chip for output control and acquisition. Each remote unit can have 
up to four loop control chips, each of which can directly accept two 
16-bit data paths. The system is designed for continuous high- 
throughput data streams with many parallel tasks and data paths. 
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MAIN INTERFACE BOARD 



The main interface board outlined in Fig. 2 consists 
of five main components. The current prototype board 
fits on a full-length PCI card. Optimization of the PCI 
interface chip and discrete gates will reduce the size of 
the main card to a half-size card, or the components of 
the remote board can be added to a full-length single 
board. 

A PCI interface chip controls DMA transfers to the 
host and maps memory locations for I/O ports, data 
buffer memory, and DSP memory. A 54-bit bus switch 
is responsible for swapping the data and address busses 
of the PCI interface chip and DSP with two banks of 
data buffer memory. The bus switch introduces only 0.3 
ns of signal delay to the signal path and can switch the 
data and address busses within 7 ns. The bus switch 
allows data collection to one bank while the host is 
reading data from the other; when the data collection 
buffer is full, the banks are swapped, and continuous 
data transfers are possible. Buffer memory is 32 bits 
wide and can be upgraded from 64K words to 16M 
words. Typically, 1 to 5 s worth of data is stored in a 
buffer before switching occurs. The host can also re- 
quest a buffer switch whenever it is ready for more 
data; thus, higher-performance computer systems have 
the capability to display more of the data with less 
buffer pipeline delay. 

Several standard computer interfaces are available 
to transfer data from the acquisition system to the host 
computer. Newer boards use the PCI interface with a 
32-bit bus and 33-MHz clock for a maximum transfer 
rate of 132 MB/s. More recent versions of the PCI inter- 
face are similar to the original PCI standard, but with 
enhancements. The new PCI-X and Compact-PCI in- 
terfaces have a 64-bit bus, and can have a clock rate of 
66 or 133 MHz for a maximum transfer rate of 1 GB/ 
s. PCI-X is currently being developed for Desktop PCs, 
while the Compact-PCI interface is currently imple- 
mented in rack-mount systems. 

For high data acquisition rates, cabling between the 
acquisition system and the host computer becomes a 
major issue. Faster data rates over copper wire cable 
require wider data busses and/or higher transfer clock 
rates. Wider cables result in a bulky system while 
higher data rates often result in impedance issues that 
can degrade the reliability of the data transfer. For 
these reasons, we believe optical fiber will be the trans- 
fer medium of choice in the near future. Already, many 



computer subsystems have the capability of transfer- 
ring data through optical fiber at high rates. For exam- 
ple, fiber-link hard drive controllers and high-speed fi- 
ber ethernet controllers are capable of transferring 
greater than 10 GB/s. 



OPTIONAL DSP CHIPS 



The acquisition hardware can connect directly to the 
bus switch to transfer collected data; however, utiliza- 
tion of the Analog Devices SHARC digital signal proc- 
essing chip provides many advantages. First, parallel 
tasking within the DSP chip allows preprocessing of 
incoming data such as digital filtering and image analy- 
sis in parallel with data transfer. The SHARC DSP 
provides two data move operations, two math opera- 
tions, and one compare operation in every clock cycle, 
making it the highest-performance DSP available. 
Math capabilities include integer and floating point op- 
erations. Optimized instructions for Fourier analysis 
allow real-time digital filtering of electrophysiological 
channels. The SHARC DSP chips also provide six high- 
speed Link ports for transferring data to and from re- 
mote acquisition boards up to 10 ft away from the host 
system. Such an arrangement allows flexible modular 
configuration of acquisition components and simplifies 
the design of complex systems. SHARC chips are among 
the least expensive DSPs available, ranging in price 
from $30 to $300 depending on clock speed, internal 
SRAM, and Link port capabilities. 



REMOTE BOARD 



Up to six remote boards can be connected to the main 
board through each of six Link ports available on the 
DSP chips. The remote board consists of analog-to-digi- 
tal subsystems and a DSP chip for data transfers to 
the main board and optional data preprocessing. Up to 
four FPGA-based loop control chips can be connected 
to the DSP chip. Each loop control chip has 64 bits of 
digital output for process control, 16 bits of timed inter- 
val triggers, and two dedicated 16-bit input registers. 
Each loop control chip can independently control and 
acquire data from a variety of sources; for example, a 
single remote board may have two loop control chips, 
one for controlling and acquiring from a CCD camera 
and the other for capturing electrophysiological data. 
Remote boards can be up to 10 ft away from the host 
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system and allow flexibility in modular acquisition sta- 
tions that report back to the main host computer. 



LOOP CONTROL CHIP 



The internal structure of the loop control chip is out- 
lined in Fig. 3. An FPGA contains a simplified micro- 
processor core that executes instructions stored in an 
external 64K by 32-bit SRAM bank. In the current sys- 
tem, we use the QL4036-2 (Quicklogic, CA) device which 



contains 36K gates and 12K built-in SRAM. We chose 
Quicklogic devices over other manufacturers such as 
Xilinx and Altera for two main reasons. Early proto- 
types used Xilinx and Altera devices; unfortunately 
these FPGAs were too slow for adequate real-time, non- 
pipelined control logic (10 MHz). Also, logic blocks 
within Xilinx and Altera devices were not optimized 
for high-speed counters and high-level multiplexing. 
Quicklogic devices provided the fastest logic implemen- 
tation of any of the devices we evaluated (80-100 MHz) 
and required the fewest logic blocks to implement the 
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FIG. 3. The internal structure of the loop control chip contains several main components. Communication to the SHARC DSP is achieved 
through a 32-bit data I/O bus and interrupt generator (IRQ). Four 32-bit registers control components within I he chip. The control register 
manages various processes of the chip, including programming of the instruction SRAM and setting up of the IRQ mask. The instruction 
register connects directly to the instruction SRAM I/O bus and is used to I ransl'er itisi run ions, ['he rose! register is responsible for clearing 
flags within the chip, such as "FIFO full" and IRQ codes. The counter register is used for process counters. A 32-bit FIFO (first-in, first- 
out) memory bank is used to buffer incoming data, and a 32-bit status register can inform the SHARC DSP of the status of internal variables 
such as flag states and error conditions. The instruction SRAM I/O and SRAM address component soft he chip a re responsible for programming 
tlx- microcode instructions into 64K X 32-bit-wide external memory. During operation of the chip, the SRAM address is incremented after 
each cycle, unless a jump location is requested. The 32-bit instruction word is divided into two parallel 16-bit paths. The 16-bit data path 
is used to output control signals to one of four output control registers, determine the "jump to" address, output direct ly to t he acquisition 
data stream, or program one of six 16-bit counters. The 16-bit instruction path simultaneously performs four tasks. Bits 0-3 define a flag 
condition code that determines whether a jump will occur, bits 4-7 decide where the 16-bit data word will go, bits 8-11 determine which 
input path will be used in the acquisition data stream, and bits 12-15 are four dedicated output lines for high-speed signal control. A 64- 
bit instruction counter increments once with each instruction cycle, keeps track of absolute running time, and generates a plottable "AIL" 
time code for polygraph records. Six 16-bit counters are used to keep track of the number of times an instruction is executed and also to 
decide whether a jump occurs. 
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microprocessor core. The speed advantage is mainly at 
the sacrifice of reprogrammability; Quicklogic devices 
can be programmed only once. However, we never used 
the reprogrammability of the other devices since the 
initial design was flexible enough to meet our applica- 
tion needs. 

A data bus I/O block handles all transactions between 
the internal components of the loop chip and the 
SHARC DSP. The SRAM instruction bank is pro- 
grammed via the instruction register, and instructions 
begin executing on completion of the program down- 
load. After each clock cycle, the SRAM address is incre- 
mented, and a new instruction is executed. The 32-bit 
instruction word is divided into two 16-bit paths similar 
to the Harvard Architecture, where data and instruc- 
tions appear in parallel. The 16 data bits can define 
one of four operations: (1) jump to address if conditions 
are met, (2) control data output, (3) load counter values, 
or (4) place 16-bit values within the collected data 
stream. The 16 instruction bits are divided into four 
parallel operations: (1) flag condition decode, (2) data 
output decode, (3) data input decode, and (4) four dedi- 
cated output control lines for high-speed timing signals 
such as digitizer clocks. 

The loop control chip runs at 40 MHz and allows 
parallel output, input, and control in a user-defined 
sequence with well-defined timing not possible with 
other microprocessor cores. This arrangement allows 
us to write control sequences that are downloaded into 
the SRAM chip to control a CCD camera and a variety 
of other hardware and simultaneously digitize CCD 
output and a large number of electrophysiological chan- 
nels. Digitized data are directed into one of the double- 
buffered SRAM banks. When a bank is full, it is immedi- 
ately swapped, the host is notified of a full buffer, and 
digitizing continues into the new buffer. 



CCD CAMERA 



The current CCD camera chip (TC211, Texas Instru- 
ments) has a maximum pixel readout rate of 10 Msps, 
with a maximum full-frame rate of 250 Hz. Skipping 
every other readout line increases the frame rate to 500 
Hz. Currently, the TC211 chips have a 1.5-LSB noise 
level at a maximum readout rate of 10 Msps and 12 
bits digitization. An improved camera (Felix) designed 
for us by Dalsa Inc. of Canada has the ability to read 
out the pixels at 30 Msps and has a high signal-to- 
noise ratio. The faster readout rate will increase the 
maximum frame rate to 1500 Hz and improve the tem- 
poral resolution of the data. 



DATA ARCHIVING 



We prefer to archive as much of the raw data as 
possible; however, we encountered a significant prob- 
lem in identifying storage space for long continuous 
streams of high-density data. In some circumstances, 
the data can be compressed to a certain degree, saving 
archival space, as well as reducing the load at potential 
bottlenecks. We have experimented with a number of 
compression algorithms, including byte packing, jpeg, 
and various lookup table algorithms. In general, the 
host spent more time compressing the data than was 
gained by the effort. Typically, a 10% reduction in data 
was achieved at best. Image data, if black level and 
gain are properly optimized, are not very compressible 
except by lossy techniques. In our experience, any type 
of lossy compression introduces artifacts that appear 
on images processed by subtraction or ratio routines. 



FILE FORMAT 



The file format should be easy to use and contain 
complete embedded documentation, so that someone 
coming back to the data 1 00 years later could interpret 
the data stream. There trade-offs to be considered in 
the file format. The easiest data to access are in some 
kind of raw form, for example, a raw binary sequence 
of digitized values. However, the program interpreting 
the data would be lost if you changed the number of 
input channels or the acquisition rate midstream. 
Someone perusing the data 100 years (or 100 days) later 
would be at a loss to interpret the data. On the other 
extreme, a data set with too much descriptive informa- 
tion is wasteful, bulky, difficult, and slow to parse. 

A good compromise is to have a dataset with occa- 
sional headers describing the data to follow, with 
enough information to let the parser know about the 
format of the data. Such a file format would also have 
to be flexible enough to handle a great variety of data 
types, specifications, and styles. We chose to use the 
IFFPHYS (Interleaved File Format for Physiological 
Data, D. Sirag) format due to its consideration of all 
these features. 

IFFPHYS is a standard interchange format for meas- 
ured data from a wide variety of data acquisition sys- 
tems. The data set is frequently too large to fit into 
available random access memory, and thus usually 
must be processed in blocks from a mass storage device 
with data from all channels available in each block. 
Files containing the physical data should be able to 
preserve all related information in a standard manner 
so that the data are available to a variety of analytic 
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programs. The objective of IFFPHYS is to specify a 
uniform manner to record physical data into a file and 
yet accommodate the wide variety of physical data and 
instruments with which measurements are made. This 
goal is accomplished by allowing the data to be stored 
in blocks preceded by a descriptive header about the 
content of the block. In addition to analog, data physical 
data include event and image data. The data can be 
considered as points in time in reference to a master 
timing generator, and thus data from any block can be 
easily and rapidly associated with any other block. As 
the complexity of the data increases, appropriate block 
header information is added to describe the complexity. 



ARCHIVE MEDIA 



There are a variety of archival media to choose from 
in storing data. Large hard drives are becoming so inex- 
pensive now that we sometimes store data on hard 
drives for pseudopermanent archival purposes. Tape 
technology has also been improving. In the past, 8-mm 
tape was the tape of choice. However, after a couple 
years, significant errors may appear since older 8-mm 
tape technology used CRC error checking which does 
not effectively identify some errors. For very large data 
sets, digital linear tape (DLT) tape technology is the 
least expensive and most reliable per MB and can sus- 
tain 5 MB/s data rates. Redundant arrays of inexpen- 
sive disk (RAID) technology allows multiple hard drives 
or tape drives to be used in parallel, allowing data trans- 
fer at the full rate of the interface bus. For our experi- 
ments having less than 5 MB/s throughput, we usually 
store data directly onto tape media. For higher data 
rates, we use RAID tape arrays, or fast hard drives. 

Once written to the archive device, the data no longer 
need to be accessed from the storage device. The incom- 
ing data stream is actually split into two separate 
streams. One high-performance stream goes directly to 
the archive media for reliable storage, and the other 
stream is directed to the various computer processes 
that display and analyze the data on-line (see Fig. 5). 



REAL-TIME CONTROL 



During a typical experiment, the investigator often 
needs to control parameters during data acquisition. A 
stimulus signal may be generated at specific times dur- 
ing the experiment and feedback may be given in re- 
sponse to certain conditions. For optimal video digitiz- 
ing, the acquisition system needs to provide the drive 
signals to the CCD camera in addition to digitizing the 



pixel data. We accomplish such control in two ways: 

(1) during the acquisition of digitized data, a parallel, 
control bus can output control signals in parallel, and 

(2) built-in DSPs can output control signals with a coarse 
time granularity for stimuli and feedback signals. 



REAL-TIME DISPLAY AND PROCESSING 

Often, data can be collected during the experiment 
and analyzed later. However, in many circumstances, 
it is useful or necessary to view some analyzed data as 
they are being collected. The level of real-time display 
and processing can be varied with the capabilities of 
the host computer system. Software that performs such 
tasks at a variable "gear ratio" work best for such proce- 
dures. This approach defines a minimum data unit, 
such as a data buffer, video frame, or electrophysiologi- 
cal sample, and assigns skip values for each data unit 
in the stream. The highest priority is to archive the 
incoming data buffers. Software must accept new data 
buffers as soon as they are available, and archive them. 
Computer processor time available between full buffers 
can be used for data transmission to other processors, 
data display, and real-time processing. Much like cogs 
on a gear, data can be displayed at regular intervals 
during the period between full buffers. If the processor 
has a lot of time, much of the data can be displayed. If 
there is not much time, more of the data are skipped. 

It is important to display data in a regular interpo- 
lated fashion so that the data display does not look 
"jerky" or distorted in time; to achieve this end, it is 
important to skip regular integral periods of time. For 
example, if the current display setting is set to show 
every frame of a video buffer, and a full buffer arrives 
before the full sequence is displayed, then the skip value 
is incremented, and the program displays every other 
frame for the next buffer. To make the images more 
evenly displayed across time, the average time between 
full buffers can be divided by the number of images to 
display to calculate a time delay factor between images. 
For truly multithreaded operating systems, it is im- 
portant that the process goes to sleep during this delay 
time. Other critical functions can be performed during 
the delay, such as archiving, processing, and data trans- 
mission to other processes. 



INTEGRATION OF DATA FROM DIFFERENT 
MODALITIES 



One of the most difficult technical challenges involves 
integration of data from different sources that need 
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to be accurately synchronized in time. This includes 
multiple sources of the same kind of data (i.e., electro- 
physiological data) or different kinds of data, such as 
video images and electrophysiology, which need to be 
time synchronized. Most systems have a difficult time 
with this procedure, especially for high-rate data 
streams, because data arrive from different sources 
with different acquisition clocks. We control acquisition 
of all data types with the same hardware, thus easing 
timing and synchronization. Data arrive at the host in 
the form of packets with header information about the 
type of data and when it was collected within the 



stream. The IFFPHYS file format greatly simplifies 
interpretation and archiving of such data. 



HOST COMPUTER AND USER INTERFACE 

We find that the Linux operating system serves well 
as an operating system for advanced data acquisition. 
Not only is it free, robust, and powerful, but it can 
handle many processes quite efficiently with rapid con- 
text switching, so that no data buffers are missed. The 
control software is written using the "C" programming 





FIG. 4. The user interface for the acquisition and control software uses at least nine basic pop-up windows for the X-Windows interface. 
Each pop-up window represents an independent process task that can reside within the main host computer, another CPU within the host 
computer, or other computer on the network. Each pop-up window requests data from the data collection server through internet socket 
protocols. The Main Sever Window (a) is used to initiate other tasks. The Record Window (b) is used to control and display the archive 
process. The Histogram Window (. I displays pixel intensity histogram profiles for each displayed image. The Preliminary Window (d) plois 
specific types of analysis; in this case, the average image intensity is plotted. Acquired images can be displayed in many forms in the Image 
Window (e) . Electrophysiological channels are plotted in polygraph form in the Analog Window (f) . Each analog channel can be displayed 
separately as shown in the Channel 1 Window (g). More sophisticated types of on-line analysis can be added, such as the Time Triggered 
Average Window (h). which configures analysis parameters and displays image sequem «>s and I races. The Loop Control Window (i) is used 
to program the loop control chip and specify data acquisition parameters. 
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language and the X-Windows interface is used for creat- 
ing buttons and widgets to allow the user to interact 
with all aspects of the program. Separate windows pop 
up for different aspects of the acquisition; for example, 
a button on the main window will bring up a new win- 
dow to display electrophysiological traces in polygraph 
form, with display rate and amplitude controls for all 
channels (Fig. 4). The user interface was implemented 
using the "Lesstif ' libraries available for Linux, a free 
version of the popular "Motif window interface. 

Elements within the user interface are easily modi- 
fied and appended as they represent separate programs 
that operate independently as separate process tasks 
and request data from the main collection server task 
through internet socket protocols. Thus, to add a fea- 
ture or have multiple views of the same data, the user 
can spawn another process to access the data. Since 
the data are in IFFPHYS format, it is easy to parse the 
data stream for data relevant to the particular window. 
Since the user interface uses X-Windows, the entire 
package is portable across other UNIX platforms in- 
cluding other high-performance hardware platforms 
(Sun, SGI, HP). This arrangement allows maximum 
flexibility for performance enhancement at minimal 
cost. Continued improvements are made with minimal 
effort and flexibility through network interfaces for dis- 
play and modular multitasking in an asynchronous par- 
allel computing environment. 

The system provides solutions for on-line analysis 



and mass storage of continuous data streams through 
the use of parallel tasking and high-speed network con- 
nections. The software collects and stores data; displays 
images, electrophysiology, and histograms; and per- 
forms a variety of simple-to-complex on-line analyses 
concurrently. Sophisticated post hoc analysis routines 
use predefined parameters to sort through images and 
create files for statistical comparison and display. 



EVOKED LIGHT SCATTERING CHANGES FROM 
THE RAT DORSAL BRAINSTEM 



We have imaged fast optical changes associated with 
evoked neural activation in the rat dorsal brainstem. 
The work employed an endoscopic imager we developed 
and the acquisition system described above. General 
surgical and physiological preparation methods have 
been described previously (15). Twenty-five male rats 
(250-450 g) were used in separate experiments, typi- 
cally one per day. Animals were anesthetized with ure- 
thane (1.3 g/kg), and the femoral artery and vein were 
cannulated. A tracheotomy was performed for artificial 
ventilation, and the right cervical vagus nerve bundle 
was dissected to identify and isolate the vagus (VN), 
aortic nerve (AN), and superior laryngeal nerve (SLN) 
from the surrounding tissue. Animals were placed in a 
stereotaxic apparatus and suspended from a cervical 
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FIG. 5. In these experiments, one remote board is responsible for controlling the CCD camera and stimulus generator, while acquiring 
electrophysiological data and CCD images. The remote board sends the raw frame sequences and physiology to the host computer through 
the interface board. The host computer splits the data stream into two duplicate streams. One stream is directed to the mass storage device; 
the other stream is used for on-line analysis, time-triggered averaging, display. 
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vertebra. The AN, VN, and SLN were placed intact on 
bipolar stimulating hook electrodes. 

The endoscope probe, described in detail elsewhere 
(12), was placed on the brainstem surface overlying 
the NTS (Fig. 5). A series of optical fibers around the 
perimeter of the probe illuminated the tissue. Backseat - 
tered light was collected using a 3-mm-diameter gradi- 
ent index (GRIN) lens (Gradient Index Corp., Rochester, 
NY). An image from beyond the end of the probe was 
focused through a microscope objective lens onto a CCD 
camera (TC245, Texas Instruments, Dallas, TX). Cam- 
era output was sampled by a 12-bit digitizer system at 
192 X 165 pixels, 100 Hz, and simultaneously collected 
16 physiological signals (1 kHz each channel), including 
the electrocardiogram (EKG), tracheal pressure, blood 
pressure, and end-tidal C0 2 . A fine wire was inserted 
under the probe for recording evoked responses to nerve 
shocks. The aggregate data rate was 6.5 MB/s. 

To increase sensitivity of very small changes in light 
intensity, we increased the effective dynamic range of 
the video digitizing system through judicious selection 
of black level and gain on the video amplifiers. The 
black level was first set such that a digital value of 
"64" represented the dimmest pixel in the image, and 
a digital value of "4032" represented the brightest pixel 
in the image. This procedure increased our ability to 
detect intensity changes by a factor of 6.7 (16). One 
disadvantage of this procedure, however, is that it re- 
quires an extra calibration step to recover absolute in- 
tensity values. 

After a 30-min stabilization period, the tissue was 
illuminated with 780-nm monochromatic light from a 
laser diode array through plastic optical fibers arranged 
around the perimeter of the GRIN lens. The depth of 
focus was initially adjusted to 400 /mi below the surface. 
Backscattered light images from the NTS were digitized 
during shocks to different nerves of the vagal bundle: 
AN, VN, or SLN stimuli were controlled by the acquisi- 
tion system. Each channel was set to an amplitude 
twice that required to evoke a reflexive change in heart 
rate and arterial pressure in response to a 1-s, 100-Hz 
pulse train. The stimuli were given in random phase 
relative to the cardiac and ventilatory cycles (typically 
40 fiA, 0.1 -ms single pulse, 1- to 2-s random interval). 
Camera frames were digitized continuously (100 fps) 
along with physiological signals (1 kHz per channel) 
and multiplexed into the IFFPHYS file format. 

The acquisition computer and software displayed im- 
ages and physiological signals in real time as well as 
image sequences averaged relative to the stimulator 
pretrigger pulse. The computer also stored images and 
physiological records on a mass storage device for subse- 
quent analysis. Image sequences were generated by 
subtracting the average of nine prestimulus images 



from each of 90 poststimulus images. An average re- 
sponse was obtained by averaging 400 to 1000 se- 
quences collected under identical conditions. Images in 
the average sequence were pseudocolored for display 
such that cool colors (blue to purple) represented in- 
creased light reflectance from the tissue (typically cor- 
related with decreased neural activity), warm colors 
(yellow to red) represented decreased light reflectance 
from the tissue (associated with increased neural activ- 
ity), and green represented no significant change from 
the first image. For comparison of neural electrical pat- 
terns with hemodynamic effects, image sequences were 
also generated using the cardiac R-wave as the trigger 
and correlated on a pixel-by-pixel basis to the blood 
pressure temporal signal. 



RESULTS 



Shocks to each nerve elicited an electrical presynaptic 
population spike 30 ms after the stimulus and a post- 
synaptic population evoked potential that peaked 80 
ms after the stimulus. Electrical stimulation produced 
consistent spatiotemporal patterns of optical changes, 
which were recorded with the imaging probe. A typical 
optical response trace to nerve stimulation integrated 
over the entire image is illustrated in Fig. 6. The time 
course plot shows rapid light scattering changes that 
follow the electrical evoked potential within the imaged 
tissue. Slower temporal components match hemody- 
namic events occurring with stimulation under the 
probe. Images from the first 100 ms are illustrated in 
Fig. 7. A prominent decrease in reflected light (yellow 
to red region) appears at the bottom of the image within 




FIG. 6. Tinie-t riggerod image sequences were collected over a 1-s 
period after stimulating the vagus nerve. All pixels of the image 
were averaged into a single trance to show the time course of light 
scattering changes. The electrical evoked potential measured from a 
macrowiro placed under I he image probe is plolled In-low I he lighl 
scattering trace. 



CONTINUOUS IMAGE AND ELECTROPHYSIOLOGICAL RECORDING 



161 



three frames (30 ms) after the stimulus and fades away 
after 140 ms. 

Similar patterns were observed for all animals stud- 
ied, with differences in the spatial organization of the 
response when stimulating different nerves. Averaged 
image sequences from the same data sets, triggered by 
the cardiac R-wave instead of the stimulus pretrigger, 
were correlated to the blood pressure signal and showed 
different spatial patterns than stimulus sequences. 
However, some longer-latency optical responses showed 
clear similarities with the characteristic hemody- 
namic patterns. 



DISCUSSION 

Our needs for high performance and integration of a 
number of data modalities required that we develop 
custom data acquisition hardware to control the CCD 
camera circuitry and digitize video frames, along with 
many electrophysiological data channels. The host com- 
puter consists of one or more Intel Pentium class proces- 
sors. A fast PCI SCSI card communicates with large 
and fast hard drives and tape drives. A PCI accelerated 
display card with at least 8 MB RAM can display images 
with high speed and quality. A fast ethernet card (100 



Mbs or higher) can transfer data rapidly between com- 
puters in a multihost system. The PCI interface is used 
to connect the acquisition board to the host computer. 

The high-performance, custom digitizing electronics 
have programmable flexibility in pixel scanning param- 
eters and allow simultaneous electrophysiological ac- 
quisition for 1 to 32,000 channels. The digitizing cir- 
cuitry stores the data in a double-buffered memory 
array using an internationally recognized standard file 
format. Double buffering allows the host computer to 
process previously collected data simultaneously with 
acquisition. 

These features allow sophisticated optical mapping 
of light scattering changes associated with neural acti- 
vation from a large number of neurons at frame rates 
rapid enough to follow dynamics of the electrical re- 
sponse and to evaluate correlations with other physio- 
logical data. In our experience, on-line processing and 
visualization of data provide feedback that is essential 
for the optimal conduct of physiological experiments. 

Hardware adaptability and speed are gained through 
use of large field programmable logic arrays in the ac- 
quisition circuitry. FPGAs can be reprogrammed on- 
line for changing experimental conditions or devices, 
although our present system does not require such 
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FIG. 7. Three image sequences show the spatial and temporal dis 
I hroo branches of the vagus, superior laryngeal, and aortic nerves. I 
displayed. Each frame is separated by 10 ms; and images are pseudocolored s 
reflectance (increased activation) , and cool colors (blue to purple) rej 

significant change from prestimulus conditions. Each pixel in the acquired sequence is also correlated to the blood pressure signal (BP) and 
displayed as correlation coefficient intensities for three lag times. The blood pressure correlation images reveal vascular structure under 
the image probe. 



if scattered light changes in response to stimulating one of 
us begins just before the first image and a 100-ms period is 
ich that warm colors (yellow to red) represent decreased 
creased reflectance (decreased activity) . Green represents no 
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capabilities. We included a dedicated digital signal proc- 
essor (DSP) to assist in preprocessing the data stream 
and high-speed communication. High-performance ac- 
quisition was achieved through circuitry designed 
around the PCI bus, which is rapidly becoming the 
standard for small but powerful personal computers at 
low cost. 

We used the system for recording changes in light 
scattering occurring in synchrony with evoked poten- 
tials in the NTS and observed consistent spatiotempo- 
ral patterns in image sequences associated with dorsal 
medulla activation after nerve shock. Stimulating dif- 
ferent branches of the vagal bundle also produced differ- 
ent spatial response patterns. 

Previous imaging studies of scattered light changes 
have failed to observe the fast optical changes reported 
here. A number of technical improvements have al- 
lowed such measurements. Dark-field illumination 
methods increased our ability to detect small changes 
in scattered light (17, 18). Bright-field illumination (2) 
tends to record light reflected from the surface; only a 
small proportion of the imaged light is perturbed by 
physical processes associated with activation occurring 
below the tissue surface. Additionally, blood vessels 
dominate the tissue surface. More of the bright-field 
signal is derived from hemodynamic responses to neu- 
ral activity. Dark-field measurements record deeper 
changes, since light must first penetrate the tissue be- 
fore returning to the camera (12, 19). 

Increased frame acquisition rates allowed recording 
of optical changes more rapidly than previously re- 
ported, providing finer temporal resolution in the data 
for recording transient responses. A high-sensitivity 
CCD camera and amplifiers allowed 12-bit digitizing 
over the dynamic range of the signal, providing greater 
sensitivity to small signal changes. Placement of the 
image collecting gradient index lens in contact with 
the tissue of interest stabilized the tissue interface, 
suppressing potential movement artifacts. Focal depth 
adjustments allowed optimization of signals originating 
from deep sources. 

Since intense illumination levels may affect the tis- 
sue through heating or photodamage, final limitations 
of the CCD camera frame rate depend on the amount 
of available illumination light provided around the pe- 
rimeter of the gradient index lens or image conduit. 
However, coupling of scattered tissue light to the CCD 
sensors through our probe provides highly efficient de- 
tection of backscattered light; nearly all of the returning 
light is captured. Because very low illumination levels 
are required, we typically use standard light-emitting 
diodes as a light source which provide plenty of light 
for 1-kHz frame acquisition. 

We plan a number of improvements to the imaging 



probe to increase the spatial and temporal resolution of 
imaging studies. The probe can be configured to create 
confocal, spectrally resolved images of the tissue, for 
better spatial resolution, biochemical and physiological 
specificity, and 3D reconstruction of tissue activation 
volume. By using voltage- and calcium-sensitive dyes, 
it may be possible to identify individual neurons in- 
volved in the response, perhaps with even more direct 
correlations to electrical activation. High-speed and 
high-sensitivity optical recordings have potential for 
studying dynamic temporal and spatial activity pat- 
terns during cognitive (20) and autonomic (21) func- 
tions which may underlie high-level processing within 
local neural networks. Using near-infrared light, it may 
be possible to perform low-resolution imaging or tomog- 
raphy of cortical activation noninvasive ly (6), perhaps 
in human subjects (22). 
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